Variable-Rate Coding with Feedback for Universal Communication Systems

Classical coding schemes that rely on joint typicality (such as Slepian-Wolf coding and channel coding) assume known statistics and rely on asymptotically long sequences. However, in practice the statistics are unknown, and the sequences are of finite length n. In this finite regime, we must allow a non-zero probability of coding error and also pay a penalty by transmitting extra bits. The penalty manifests itself as redundancy in source coding and a gap to capacity in channel coding. Our contributions in this paper are two-fold. First, we characterize the penalty due to universality for any communication system that relies on joint typicality. This penalty is of the order O( √ n) bits. For the specific example of channel coding, we develop a scheme in which the gap to capacity is O( √ n) bits for a fixed . The prior art for channel coding with known statistics shows that the penalty is Ω( √ n). Therefore, we infer that the penalty needed to accomodate universality is Θ( √ n). Second, we derive the penalty incurred in variable-rate universal channel coding when we restrict the amount of feedback to a fixed number of rounds. This result is valuable for practical communication systems.